Skip to content

feat: cross-browser screenshot baseline storage (RFC 0002)#115

Merged
PAMulligan merged 11 commits into
mainfrom
111-implement-cross-browser-screenshot-baseline-storage-rfc-0002
Jul 2, 2026
Merged

feat: cross-browser screenshot baseline storage (RFC 0002)#115
PAMulligan merged 11 commits into
mainfrom
111-implement-cross-browser-screenshot-baseline-storage-rfc-0002

Conversation

@PAMulligan

Copy link
Copy Markdown
Collaborator

Implements RFC 0002 — Cross-Browser Screenshot Baseline Storage. Closes #111.

First commit records the RFC's acceptance (approved & merged as PR #112 on 2026-07-01), clearing the implementation gate.

What this delivers, by RFC phase

Phase A — make the signal real (non-blocking)

  • visualBaselines config section + strict JSON Schema, with structural checks (single cross-engine tolerance, browsers ⊆ E2E matrix, LFS only on the commit backend)
  • Provenance manifest (baselines/manifest.json): engine, Playwright version, pinned image, sha256, gitSha, capture host — scripts/lib/baseline-manifest.js (record/sync/verify)
  • scripts/cross-browser-baseline.sh|js — capture / compare / verify. Compare diffs firefox/webkit/chromium against committed same-engine baselines via visual-diff.js at 0.03
  • Fixed the mislabeled orchestration cross-browser phase: Phase 7 now runs the comparison it advertises (dispatch table, workflow skills, /build-from-figma); cross-browser-test.sh remains as an ad-hoc capture utility
  • regression-test.sh walks only its own browsers (no SKIP noise from cross-browser baselines) and, like capture-baselines.sh, syncs manifest entries when it rewrites chromium baselines

Phase B — determinism, then teeth

  • Pinned-container capture: capture.mode: "container" wraps capture in docker run <pinned image> locally (--dry-run to inspect; --local = untrusted escape hatch), and CI runs compare/capture inside container: jobs with the image read from config
  • Provenance is enforced in the comparator: policy: "warn" reports alongside pixel results; policy: "enforce" excludes flagged baselines and counts them as failures. Non-container compares are marked advisory
  • The machinery for blocking ships (blocking config + --blocking), but stays off: blocking: false and qualityGate.crossBrowserScreenshotsRequired: false are unchanged per the RFC. Flip criteria documented (≥20 clean containerized CI compares)

Phase C — the other adapters

  • storage: "lfs" + scripts/setup-baseline-lfs.sh (idempotent .gitattributes automation, forward-only; history rewrite printed as guidance only) + drift warnings both directions
  • ci-artifact backend: compares against the cross-browser-baselines artifact published from the last green main (publish job included); clean skip before the first publish
  • service backend: delegated Chromatic/Percy hand-off, token via CI secret, opt-in

CI: new .github/workflows/cross-browser-baselines.yml — PR compare (pinned container, diff artifacts, upserted PR comment), workflow_dispatch deterministic recapture → baselines PR, main artifact publish for the ci-artifact backend.

Docs: docs/regression-testing/cross-browser.md (guide incl. §11 exposure/cost table and §13 resolutions), onboarding config/architecture tables, scripts/README, root README, CLAUDE.md, implementation plan.

RFC §13 open questions — resolutions

  1. Unify regressionTesting/visualBaselines? Kept separate (per §7.1); shared baseline dir, each flow walks only its engines
  2. Blocking threshold: machinery ships now; flip after ≥20 consecutive clean containerized compares (documented)
  3. Image pinning: explicit config pin is authoritative; scripts warn on drift vs the resolved Playwright version
  4. Local firefox/webkit capture: warn-and-record (host: "local" → flagged foreign), never silently trusted; hard-fail only under policy: "enforce"
  5. LFS migration: forward-only automation; git lfs migrate import documented, never executed

Verification

  • 73 tests across 6 new/extended suites (TDD throughout — black-box CLI tests, scratch git repos for LFS, injected exec for backends); full script suite: 500/500 tests pass (the one failing suite, agent-plugin-lib.test.js, pre-exists on a clean tree on the dev machine — its shebang trips vitest's transform, same class of issue fixed here for pipeline-config.js)
  • Live end-to-end on a real app with Playwright 1.61.1 + all three engines: capture (6 baselines + manifest) → verify (foreign-host flags on local firefox/webkit, exit 1) → compare (6× PASS at 0.00%, advisory envelope) → UI mutation (6× FAIL at 7.47%/17.32%, exit 0 non-blocking, exit 1 with --blocking) → tamper detection (modified) → LFS setup in a scratch repo
  • The smoke test caught and fixed a real display bug: visual-diff.js's mismatchPct is a 0–1 ratio, and both consumers printed it as a percent (understating diffs 100×)
  • Not exercised live: an actual docker run capture (image pull ~2 GB) — the command construction is covered by --dry-run tests, and the in-container path is what CI runs

🤖 Generated with Claude Code

Paul Mulligan and others added 11 commits July 1, 2026 20:03
Maintainer approved and merged the RFC PR (#112) on 2026-07-01; per RFC
process (docs/rfcs/README.md §3) record the acceptance in the Status row,
§12 sign-off, and the RFC index. Unblocks implementation issue #111.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…storage (RFC 0002)

New strict config section selecting the baseline backend (commit |
ci-artifact | service), git-vs-LFS storage for the commit backend, the
pinned Playwright capture image, and the provenance manifest + policy.
Structural checks keep storage=lfs on the commit backend, hold
visualBaselines.threshold equal to e2e.crossBrowserDiffThreshold (one
cross-engine tolerance), and require browsers to stay within
e2e.crossBrowserBrowsers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dual-mode ESM lib implementing the RFC 0002 §7.3 manifest: record rebuilds
entries per engine from a capture envelope (sha256, engine, route,
breakpoint, host, gitSha; top-level Playwright version + pinned image),
sync lets sibling baseline writers refresh entries without touching the
envelope, and verify classifies each baseline (ok / untracked / modified /
missing-file / foreign-host) plus manifest-level version/image drift and
an envelope-match flag that marks local compares as advisory.

Drops the inert shebang from lib modules (here and pipeline-config.js):
they are always invoked as `node <path>`, and vitest's SSR transform
rejects hashbangs in imported modules, which blocked direct lib imports
from tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ance verification

New cross-browser-baseline CLI (node core + thin bash wrapper) implementing
RFC 0002 Phase A for the commit backend:

- compare captures current screenshots (or takes --current-dir) and diffs
  them per-baseline against the committed firefox/webkit/chromium baselines
  via visual-diff.js at the 0.03 cross-engine threshold, writing a report
  with per-baseline provenance status
- provenance verification runs before diffing: warn policy reports
  stale/foreign/untracked baselines alongside pixel results; enforce policy
  excludes flagged baselines from diffing and counts them as failures
  instead of emitting false pixel diffs
- compares from a non-container envelope are marked advisory
- blocking stays config-driven (visualBaselines.blocking, default false)
  with a --blocking override, so Phase A ships non-blocking per the RFC
- capture records the manifest envelope; local firefox/webkit captures are
  permitted but recorded host=local and flagged as foreign (RFC §13 Q4:
  warn-and-record, never silently trusted)
- backends live behind the RFC §8 adapter contract; commit is implemented,
  ci-artifact/service arrive with Phase C

Playwright is resolved dynamically (CBB_PLAYWRIGHT_DIR → cwd → script dir)
since it is provisioned per-project, with a clear install hint when absent.
Transient cross-browser screenshots/diffs/report are gitignored like the
regression equivalents.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n it advertises

The orchestration cross-browser phase was labeled "Firefox and WebKit
screenshot comparison" while its dispatch ran capture-only
cross-browser-test.sh (RFC 0002 §2). Phase 7 now dispatches
cross-browser-baseline.sh compare (committed same-engine baselines +
provenance) across the orchestration dispatch table, the figma workflow
skill, visual-qa-verification, and /build-from-figma; the phase declares
its shared dev-server resource; cross-browser-test.sh is annotated as an
ad-hoc capture utility.

Sibling baseline writers stay honest about provenance: regression-test.sh
--update-baselines and capture-baselines.sh sync manifest entries for
their engines (host=local, no-op without a manifest), and the regression
walk is restricted to regressionTesting.browsers via a shared
common_find_baselines helper so committed firefox/webkit baselines no
longer surface as SKIP noise in the chromium regression report.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…baselines

capture.mode="container" (the default) now wraps capture in a docker run
of the pinned visualBaselines.capture.image: repo mounted at /work,
localhost URLs rewritten to host.docker.internal (with host-gateway
add-host), and @playwright/test installed at the image-tag version into a
scratch dir the CLI resolves via CBB_PLAYWRIGHT_DIR. The container only
writes PNGs; provenance is recorded host-side with host=container and the
host checkout's gitSha. --dry-run prints the docker command without
running it; already-in-container runs (CI `container:` jobs) short-circuit
the wrapper; --local remains the untrusted escape hatch; an image tag
without a parseable Playwright version is rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the determinism gap for CI capture (RFC 0002 Phase B): a prep job
reads visualBaselines from pipeline.config.json so the pinned Playwright
image stays config-driven, then

- PRs compare current screenshots against stored baselines inside that
  container (provenance-checked; exit code honors visualBaselines.blocking)
  with diff artifacts and an upserted PR comment
- workflow_dispatch recaptures baselines deterministically and opens a
  reviewable PR with the refreshed PNGs + manifest
- pushes to main publish the cross-browser-baselines artifact when the
  ci-artifact backend is selected (last-green-main source)

All jobs guard on baselines/app existing, mirroring the visual-regression
job (Aurelius itself carries no app under visual test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-baseline-lfs.sh implements the storage="lfs" flag (RFC 0002 §6.1,
§13 Q5 → forward-only): idempotently appends the baselines LFS filter to
.gitattributes and runs `git lfs install --local` in the CURRENT repo
(downstream apps run it in theirs). It refuses unless
visualBaselines.storage is "lfs" (--force overrides), supports --dry-run,
and prints the destructive `git lfs migrate import` history rewrite as
guidance only — never executes it.

cross-browser-baseline capture/compare now warn on storage↔.gitattributes
drift in both directions (lfs configured without a filter, or a filter
left behind on storage=git).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dapter contract

Completes the RFC 0002 §8 adapter matrix:

- ci-artifact: fetch() locates the last green main run of
  cross-browser-baselines.yml via gh and downloads the published
  cross-browser-baselines artifact (baseline root + manifest override);
  an absent artifact is a clean "no published baselines yet" skip, and a
  missing gh CLI is a clear BackendError
- service: delegated to Chromatic or Percy — compare validates the
  configured project-token env var, then hands off to the provider CLI
  (chromatic honors blocking via --exit-zero-on-changes; percy gets a
  snapshots file generated from the configured routes)

compare now consumes the full adapter surface: delegated backends
short-circuit the pixel path, and fetch() may redirect the baseline root
and manifest to the downloaded artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New docs/regression-testing/cross-browser.md covers the full RFC 0002
flow: determinism rationale, provenance statuses and policies, backend
matrix (commit/ci-artifact/service with the §11 exposure-and-cost
tradeoffs), LFS adoption (forward-only, migration guidance), CI jobs, the
blocking-flip criteria (§13 Q2: ≥20 clean containerized compares), and
troubleshooting.

Reference sweep: regression-testing README (shared baseline dir + manifest
sync note), onboarding architecture/pipeline-configuration tables
(visualBaselines key reference), scripts/README, root README script trees,
error-recovery Phase 7 row, and CLAUDE.md (script list, Phase 7 diagram
line, features bullet, key artifacts, honest script count 40→55). Commits
the implementation plan for the branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Caught during an end-to-end smoke against a live app: visual-diff.js's
JSON field mismatchPct is a 0-1 RATIO despite the name, and both
consumers appended "%" to it — understating diffs 100x (a 7.47% pixel
change printed as "0.0747%"). cross-browser-baseline now carries the
value as mismatchRatio and formats real percentages in its log lines and
report; regression-test.sh converts before printing. Also applies
repo Prettier style to the new files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@PAMulligan PAMulligan linked an issue Jul 2, 2026 that may be closed by this pull request
11 tasks
@PAMulligan
PAMulligan merged commit a06be2b into main Jul 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement cross-browser screenshot baseline storage (RFC 0002)

1 participant